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1.0  EXECUTIVE  SUMMARY 


1.1  BACKGROUND 

The  Chemistry  Division  of  the  Naval  Research  Laboratory  (NRL)  has  participated  in  several 
programs  funded  by  the  Strategic  Environmental  Research  and  Development  Program  (SERDP) 
and  the  Environmental  Security  Technology  Certification  Program  (ESTCP)  whose  goal  has 
been  to  enhance  the  classification  ability  of  the  Multisensor  Towed  Array  Detection  System 
(MTADS).  The  process  has  been  based  on  making  use  of  both  the  location  infonnation  inherent 
in  an  item’s  magnetometry  response  and  the  shape  and  size  infonnation  inherent  in  the  response 
to  electromagnetic  induction  (EMI)  sensors.  To  date,  most  of  these  systems  have  used  time- 
domain  EMI  (TEM)  sensors  with  the  notable  exception  of  the  Geophex  GEM-3  frequency- 
domain  sensor.  In  past  efforts,  classification  perfonnance  has  been  limited  by  both  the 
infonnation  available  from  the  EMI  sensors  and  by  signal-to-noise  limitations.  Three  of  the 
largest  noise  tenns  are  inherent  sensor  noise,  motion-induced  noise,  and  sensor  location 
uncertainty. 

The  three  most  successful  demonstrations  to  date  of  EMI-based  discrimination  all  involved  cued 
detection  with  gridded  collection  of  EMI  data.  The  success  of  the  gridded  data  collections  was 
due  to  the  combination  of  minimal  location  uncertainty,  no  motion-induced  noise,  and  sufficient 
signal-to-noise  ratio  (SNR).  The  downside  of  the  implementations  previously  demonstrated  is 
that  they  were  relatively  slow  and  inefficient,  especially  on  a  large  site.  The  time-domain 
electromagnetic  MTADS  (TEMTADS)  array  was  designed  to  combine  the  data  quality 
advantages  of  a  gridded  survey  with  the  coverage  efficiencies  of  a  vehicular  system.  The  design 
goal  of  this  system  was  to  collect  data  equal,  if  not  better,  in  quality  to  the  best  gridded  surveys 
(the  relative  position  and  orientation  of  the  sensors  being  known  better  for  gridded  data)  while 
prosecuting  many  more  targets  each  field  day. 

1.2  OBJECTIVES  OF  THE  DEMONSTRATION 

The  objective  of  this  demonstration  was  to  validate  the  perfonnance  of  the  TEMTADS  platfonn 
through  two  blind  tests.  The  TEMTADS  was  evaluated  in  tenns  of  both  classification 
performance  (e.g.,  false  alarm  rejection)  and  appropriateness  for  fielding  (i.e.  production  rate, 
usability,  etc.).  The  first  demonstration  was  conducted  at  the  Aberdeen  Proving  Ground  (APG) 
Standardized  Unexploded  Ordnance  (UXO)  Test  Site.  The  second  demonstration  was  conducted 
as  part  of  the  ESTCP  UXO  Classification  Study  at  the  fonner  Camp  San  Luis  Obispo  (SLO).  At 
each  demonstration,  the  site  had  been  blind  seeded  with  a  significant  number  of  intact,  inert 
UXO  types  to  challenge  UXO  classification  systems  and  methodologies. 

1.3  DEMONSTRATION  RESULTS 

The  raw  signature  data  from  the  TEMTADS  array  reflect  details  of  the  sensor/target  geometry  as 
well  as  inherent  EMI  response  characteristics  of  the  targets  themselves.  In  order  to  separate  out 
the  intrinsic  target  response  properties  from  sensor/target  geometry  effects,  the  measured 
signature  is  inverted  to  estimate  principal  axis  magnetic  polarizabilities  using  a  standard  induced 
dipole  response  model.  The  perfonnance  metrics  used  to  monitor  the  success  of  the  technology 
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relate  to  production  rate,  accuracy  of  inverted  features,  analysis  time,  correct  classification,  and 
ease  of  use. 

The  system  was  able  to  consistently  interrogate  125  or  more  cued  targets  per  day.  The  analysis, 
which  required  roughly  15  minutes  per  target,  resulted  in  a  false  alarm  reduction  by  over  50% 
with  95%  correct  identification  of  munitions.  The  average  error  in  predicted  location  was  less 
than  10  cm  in  northing  and  easting,  and  the  average  error  in  depth  estimation  was  less  than  5  cm 
for  non-overlapping  targets  with  reasonable  SNR.  Qualitatively,  the  TEMTADS  array  was 
found  to  be  easy  to  use  and  proved  to  be  a  robust  and  reliable  sensor  platfonn. 

1.4  IMPLEMENTATION  ISSUES 

Implementation  issues  for  this  system  and  technology  fall  into  two  categories:  operational 
concerns  and  data  quality/analysis  issues.  In  terms  of  operational  concerns,  the  TEMTADS  as 
implemented  is  a  large,  vehicle-towed  system  that  operates  best  in  large,  open  areas.  As  seen  at 
the  fonner  Camp  SLO  demonstration,  it  is  possible  to  operate  the  system  in  rocky  terrain  with 
grades  approaching  20%  but  at  reduced  operating  capacity  and  increased  system  wear.  Smaller 
versions  of  the  system  are  currently  under  development  under  several  ESTCP  and  SERDP 
projects  to  address  these  concerns.  The  goal  is  to  design  and  field  units  more  amenable  to 
operation  in  more  confined  terrain  and  topology  and  smaller  tow  vehicles,  man-portable  and 
handheld  operation.  Another  serious  limitation  is  anomaly  density.  For  all  sensors,  there  is  a 
limiting  anomaly  density  above  which  the  response  of  individual  targets  cannot  be  separated. 
We  have  chosen  relatively  small  sensors  for  this  array,  which  should  help  with  this  problem,  but 
we  cannot  eliminate  it.  Anomaly  densities  of  300  anomalies/acre  or  higher  would  limit  the 
applicability  of  this  system  as  more  than  20%  of  the  anomalies  would  have  another  anomaly 
within  a  meter.  In  terms  of  data  quality,  one  pays  a  small  penalty  in  signal  amplitude  to  use  the 
smaller  TEMTADS  sensor  coil  as  compared  to  other  systems  such  as  a  high-power  EM61  Mkll. 
The  dramatically  improved  electrical  performance  of  these  new  sensors  helps  counterbalance  this 
issue,  particularly  in  the  ability  to  perform  better  averaging  (or  stacking).  However,  one  needs  to 
have  in  place  robust  data  quality  control  (QC)  techniques  to  know  when  to  employ  these 
capabilities  in  an  efficient  manner. 


2 


2.0  INTRODUCTION 


2.1  BACKGROUND 

UXO  detection  and  remediation  is  a  high  priority  triservice  requirement.  As  the  Defense  Science 
Board  recently  wrote,  “Today’s  UXO  cleanup  problem  is  massive  in  scale  with  some  10  million 
acres  of  land  involved.  Estimated  cleanup  costs  are  uncertain  but  are  clearly  tens  of  billions  of 
dollars.  This  cost  is  driven  by  the  digging  of  holes  in  which  no  UXOs  are  present.  The 
instruments  used  to  detect  UXOs  (generally  located  underground)  produce  many  false  alarms — 
i.e.,  detections  from  scrap  metal  or  other  foreign  or  natural  objects — for  every  detection  of  a  real 
unexploded  munition  found.”  [1] 

There  is  general  agreement  that  the  best  solution  to  the  false  alarm  problem  involves  the  use  of 
EMI  sensors  which,  in  principle,  allow  the  extraction  of  target  shape  parameters  in  addition  to  a 
size  and  depth  estimate.  We  and  others  have  fielded  systems  with  either  time-domain  or 
frequency-domain  EMI  sensors  with  the  goal  of  extracting  reliable  target  shape  parameters  and 
thus  improving  the  classification  capability  of  our  surveys.  In  practice,  the  classification  ability 
of  these  sensors  has  been  limited  by  signal-to-noise  limitations.  Three  of  the  largest  noise  terms 
are  inherent  sensor  noise,  motion-induced  noise,  and  sensor  location  uncertainty. 

The  three  most  successful  demonstrations  of  EMI-based  classification  all  involved  cued 
detection  with  gridded  collection  of  EMI  data  [2,3,4].  The  success  of  the  gridded  data 
collections  was  due  to  the  combination  of  minimal  location  uncertainty,  no  motion-induced 
noise,  and  sufficient  SNR.  The  downside  of  the  implementations  previously  demonstrated  is  that 
they  were  relatively  slow  and  inefficient,  especially  on  a  large  site.  We  have  constructed  an  EMI 
sensor  array  that  combines  the  classification  ability  of  a  gridded  survey  with  the  coverage 
efficiency  of  a  vehicular  array.  By  coming  to  a  stop  over  each  target  to  be  investigated,  we  are 
able  to  obtain  all  the  benefits  of  a  gridded  survey  (negligible  relative  sensor  location  uncertainty, 
no  motion-induced  noise,  and  high  SNR),  while  moving  rapidly  to  the  next  target  with  no  setup 
required  gives  us  the  coverage  efficiency  required  for  practical  success. 

2.2  OBJECTIVE  OF  THE  DEMONSTRATION 

The  objective  of  this  demonstration  was  to  validate  the  technology  through  a  series  of  blind  test 
demonstrations.  We  conducted  a  shake-down  demonstration  of  the  technology  at  our  Blossom 
Point,  MD,  field  site,  but  a  blind  test  is  the  only  true  measure  of  system  perfonnance.  The  first 
demonstration  was  conducted  at  the  APG  Standardized  UXO  Test  Site.  The  second 
demonstration  was  conducted  as  part  of  the  ESTCP  UXO  Classification  Study  at  the  former 
Camp  SLO.  At  each  demonstration,  the  site  had  been  blind-seeded  with  a  significant  number  of 
intact,  inert  UXO  types  to  challenge  UXO  classification  systems  and  methodologies. 
Demonstration  scoring  was  conducted  by  third  parties  to  maintain  the  integrity  of  the  ground 
truth  and  to  provide  an  unbiased  evaluation. 

2.3  REGULATORY  DRIVERS 

Stakeholder  acceptance  of  the  use  of  classification  techniques  on  real  sites  will  require 
demonstration  that  these  techniques  can  be  deployed  efficiently  and  with  high  probability  of 
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discrimination.  The  first  step  in  this  process  was  to  demonstrate  acceptable  perfonnance  on 
synthetic  test  sites  such  as  that  at  Aberdeen.  As  a  second  step,  demonstration  on  a  care  fully 
prepared  and  blind-seeded  live  site  presented  a  more  real-world  scenario  while  providing 
sufficiently  complete  validation  data  to  accurately  determine  system  performance.  After  these 
hurdles  have  been  passed,  successful  demonstration  at  live  sites  will  further  facilitate  regulatory 
acceptance  of  the  UXO  classification  technology  and  methodology. 
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3.0 


TECHNOLOGY 


3.1  TECHNOLOGY  DESCRIPTION 

3.1.1  EMI  Sensors 

The  EMI  sensor  used  in  the  TEMTADS  array  is  based  on  the  Navy-funded  advanced  ordnance 
locator  (AOL),  developed  by  G&G  Sciences.  The  AOL  consists  of  three  transmit  coils  arranged 
in  a  1  m  cube;  we  have  adopted  the  transmit  (Tx)  and  receive  (Rx)  subsystems  of  this  sensor 
directly,  converted  to  a  5x5  array  of  35  cm  sensors,  and  made  minor  modifications  to  the  control 
and  data  acquisition  computer  to  make  it  compatible  with  our  deployment  scheme. 

A  photograph  of  an  individual  sensor  element  under  construction  is  shown  in  the  left  panel  of 
Figure  1.  The  transmit  coil  is  wound  around  the  outer  portion  of  the  form  and  is  35  cm  on  a  side. 
The  25  cm  receive  coil  is  wound  around  the  inner  part  of  the  form,  which  is  re-inserted  into  the 
outer  portion.  An  assembled  sensor  with  the  top  and  bottom  caps  used  to  locate  the  sensor  in  the 
array  is  shown  in  the  right  panel  of  Figure  1. 


Figure  1.  Construction  details  of  an  individual  EMI  sensor  (left  panel)  and  the  assembled 

sensor  with  end  caps  attached  (right  panel). 


Decay  data  are  collected  with  a  500  kHz  sample  rate  until  25  ms  after  turn  off  of  the  excitation 
pulse.  This  results  in  a  raw  decay  of  12,500  points,  too  many  to  be  practical.  These  raw  decay 
measurements  are  grouped  into  115  logarithmically  spaced  “gates,”  whose  center  times  are 
between  42  ps  to  25  ms  with  5%  widths  and  are  saved  to  disk.  Examples  of  the  measured 
transmit  pulse,  raw  decay,  and  gated  decay  are  shown  in  Figure  2. 
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Timp  fm.<^ 

Figure  2.  Measured  transmit  current  (on  time  upper  panel,  off  time  second  panel), 
full  measured  signal  decay  (third  panel),  and  gated  decay  (fourth  panel), 

as  discussed  in  the  text. 


3.1.2  Sensor  Array 

The  25  individual  sensors  are  arranged  in  a  5x5  array,  as  shown  in  Figure  3.  The  center-to- 
center  distance  is  40  cm  yielding  a  2  mx2  m  array.  Also  shown  in  Figure  3  is  the  position  of  the 
three  Global  Positioning  System  (GPS)  antennae  that  are  used  to  determine  the  location  and 
orientation  of  the  array  for  each  cued  measurement.  A  picture  of  the  array  mounted  on  the 
MTADS  EMI  sensor  platform  is  shown  in  Figure  4. 
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EM  Sensor 


GPS  Antenna 


Figure  3.  Sketch  of  the  EMI  sensor  array  showing  the  position  of  the  25  sensors  and  the 

three  GPS  antennae. 


Figure  4.  Sensor  array  mounted  on  the  MTADS  EMI  sensor  platform. 


3.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 


The  TEMTADS  array  was  designed  to  combine  the  data  quality  advantages  of  a  gridded  survey 
with  the  coverage  efficiencies  of  a  vehicular  system.  The  design  goal  of  this  system  was  to 
collect  data  equal,  if  not  better,  in  quality  to  the  best  gridded  surveys  (the  relative  position  and 
orientation  of  the  sensors  being  known  better  for  gridded  data)  while  prosecuting  many  more 
targets  each  field  day. 

There  are  obvious  limitations  to  the  use  of  this  technology.  The  array  is  a  2  m  square  so  fields 
where  the  vegetation  or  topography  interferes  with  passage  of  a  trailer  that  size  will  not  be 
amenable  to  the  use  of  the  present  array.  The  other  serious  limitation  will  be  anomaly  density. 
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For  all  sensors,  there  is  a  limiting  anomaly  density  above  which  the  response  of  individual 
targets  cannot  be  separated.  We  have  chosen  relatively  small  sensors  for  this  array,  which 
should  help  with  this  problem  but  we  cannot  eliminate  it.  Based  on  experiments  at  our  test  pit  at 
Blossom  Point,  the  results  of  this  demonstration,  and  work  done  on  the  former  Camp  Sibert  data 
sets,  anomaly  densities  of  300  anomalies/acre  or  higher  would  limit  the  applicability  of  this 
system  as  more  than  20%  of  the  anomalies  would  have  another  anomaly  within  a  meter.  For  low 
SNR  targets,  our  standard  data  acquisition  parameters  may  not  be  sufficient.  The  system 
software  has  built  in  the  capability  to  vary  the  data  acquisition  parameters  on-the-fly  based  on 
flags  in  the  target  file  and  can  be  reconfigured  manually  as  required.  One  area  in  need  of 
development  is  a  robust,  consolidated  data  collection/data  QC  methodology  for  determining 
when  there  is  a  low  SNR  anomaly  and  when  there  is  no  anomaly  present.  This  issue  is  an  area  of 
ongoing  research. 
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4.0  PERFORMANCE  OBJECTIVES 


The  TEMTADS  array  was  deployed  to  demonstrations  at  both  the  APG  Standardized  UXO  Test 
Site  and  the  ESTCP  UXO  Classification  Study  site  at  the  former  Camp  SLO.  Due  to  the  nature 
of  the  demonstrations  at  each  site  and  the  employed  scoring  methods,  separate  Perfonnance 
Objectives  were  used  for  each  demonstration,  as  documented  in  the  individual  demonstration 
reports  [5,6]  and  the  project  final  report  [7].  The  Performance  Objectives  for  the  two 
demonstrations  have  been  grouped  into  a  unified  collection  given  in  Table  1  due  to  significant 
overlap.  Please  refer  to  the  references  for  the  specific  objectives  for  each  demonstration.  Since 
the  TEMTADS  array  is  a  discrimination  technology,  the  perfonnance  objectives  focus  on  the 
second  step  of  the  UXO  survey  problem;  we  assume  that  the  anomalies  from  all  targets  of 
interest  have  been  detected  and  included  on  the  target  list  that  we  worked  from. 

Table  1.  Performance  objectives  for  the  demonstrations. 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Results 

Quantitative  Performance  Objectives 

Site  coverage 

Fraction  of  assigned 
anomalies  interrogated 

Survey  results 

100%  as  allowed  for  by 
topography/vegetation 

Yes 

Calibration  strip 
results 

System  response 
consistently  matches 
physics-based  model 

System  response 
curves 

Daily  calibration  strip 
data 

<  15%  rms  variation  in 
amplitude 

Down-track  location  ±  25cm 
All  response  values  fall 
within  bounding  curves 

No 

Reduction  of  false 
alarms 

Number  of  false  alarms 
eliminated  at  demonstrator 
operating  point 

Prioritized  dig  list 
Scoring  report  from 
APG 

Reduction  of  false  alarms  by 
>  50%  with  95%  correct 
identification  of  munitions 

Yes 

Location  accuracy 

Average  error  and  standard 
deviation  in  both  axes  for 
interrogated  items 

Estimated  location 
from  analysis 

Scoring  reports 

AN  and  AE  <  10  cm 
oN  and  aE  <  15  cm 

No 

Depth  accuracy 

Standard  deviation  in  depth 
for  interrogated  items 

Estimated  location 
from  analyses 

Ground  truth  from 
validation  effort 

ADepth  <  5  cm 
oDepth  <  10  cm 

No 

Production  rate 

Number  of  targets 
interrogated  each  day 

Log  of  field  work 

75  targets  per  day 

Yes 

Data  throughput 

Throughput  of  data  QC 
process 

Log  of  analysis  work 

All  data  QCed  on  site  and  at 
pace  with  survey 

Yes 

Analysis  time 

Average  time  required  for 
inversion  and  classification 

Log  of  analysis  work 

15  min  per  target 

Yes 

Qualitative  Performance  Objective 

Ease  of  use 

Feedback  from 
operator  on  ease  of 
use 

No  significant  operational 
issues  identified  by  operator 

Yes 

4.1  OBJECTIVE:  SITE  COVERAGE 

Each  demonstration  commenced  with  a  list  of  previously  identified  anomalies,  whose  locations 
were  detennined  using  some  other  data.  The  expectation  of  the  demonstration  was  to  gather 
cued  data  with  the  TEMTADS  system  over  each  assigned  anomaly. 
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4.1.1  Metric 


Site  coverage  is  defined  as  the  fraction  of  the  assigned  anomalies  surveyed  by  the  TEMTADS. 
Exceptions  were  made  for  topology/vegetation  interferences.  This  is  particularly  true  for 
demonstrations  where  the  footprints  of  the  detection  and  classification  systems  are  significantly 
different  (e.g.  EM61  Mkll  cart  and  TEMTADS). 

4.1.2  Data  Requirements 

The  collected  data  were  compared  to  the  original  anomaly  list.  Any  interference  was  noted  in 
the  field  log  book  as  it  occurred. 

4.1.3  Success  Criteria 

The  objective  is  considered  met  if  100%  of  the  assigned  anomalies  are  surveyed  with  the 
exception  of  anomalies  located  in  areas  that  cannot  be  surveyed  due  to  topology/vegetation 
interferences. 

4.1.4  Results 

This  objective  was  successfully  met.  All  assigned  anomalies  at  APG  were  successfully 
investigated.  At  the  fonner  Camp  SLO,  all  but  five  of  the  assigned  anomalies  were  measured. 
Failure  to  measure  these  five  anomalies  was  due  to  the  presence  of  rocks,  which  prevented  the 
operator  from  positioning  the  TEMTADS  over  the  target. 

4.2  OBJECTIVE:  CALIBRATION  STRIP  RESULTS 

This  objective  supports  that  each  sensor  system  is  in  good  working  order  and  collecting 
physically  valid  data  each  day  and  applies  only  to  the  former  Camp  SLO  demonstration.  The 
calibration  strip  was  surveyed  twice  daily.  The  peak  positive  response  of  each  emplaced  item 
from  each  run  was  compared  to  the  physics-based  response  curves  generated  prior  to  data 
collection  on  site  using  each  item  of  interest. 

4.2.1  Metric 

The  reproducibility  of  the  measured  response  of  each  sensor  system  to  the  items  of  interest  and 
the  comparison  of  the  response  to  the  response  predicted  by  the  physics-based  model  defines  this 
metric. 

4.2.2  Data  Requirements 

Response  curves  for  each  sensor/item  of  interest  pair  were  used  to  document  what  the  physics- 
based  response  of  the  system  to  the  item  should  be.  The  tabulated  peak  response  values  from 
each  survey  of  the  calibration  strip  were  used  to  demonstrate  the  reproducibility  and  validity  of 
the  sensor  readings. 
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4.2.3  Success  Criteria 


The  objective  will  be  considered  met  if  all  measured  responses  fall  within  the  range  of  physically 
possible  values  based  on  the  appropriate  response  curve.  Additionally,  the  root  mean  square 
(RMS)  variation  in  responses  should  be  less  than  15%  of  the  measured  response  and  the  down- 
track  location  of  the  anomaly  should  be  within  25  cm  of  the  corresponding  seeded  item’s  true 
location. 

4.2.4  Results 

This  objective  was  not  successfully  met  in  full.  The  measured  peak  signals  for  all  of  the 
emplaced  items  generally  fit  well  within  the  physics-based  response  bounding  curves,  with  the 
4.2-inch  mortar  and  the  shotput  results  giving  the  poorest  match,  with  a  tendency  to 
underestimate  the  peak  value.  Careful  examination  of  the  data  shows  that  this  variation  is  the 
result  of  the  shot-to-shot  precision  with  which  the  array  was  positioned  in  exactly  the  same  spot 
each  time.  Because  the  response  curves  are  generated  assuming  the  target  is  directly  below  the 
sensor,  any  offset  in  the  sensor  position  will  result  in  the  derived  peak  signal  being  smaller  than 
that  predicted  by  the  curve,  as  is  observed.  Due  to  the  large  footprint  of  the  TEMTADS  array 
and  number  of  sensor  elements  contained  in  the  array,  the  array  is  considered  to  have  been 
properly  positioned  from  a  field  work  prospective  for  a  measurement  if  the  array  center  is  within 
20  cm  of  the  target  location.  This  does  not  impact  the  values  of  the  inverted  parameters  and 
offers  the  vehicle  operator  some  flexibility  in  the  field.  It  does,  however,  affect  the  measured 
peak  signal  amplitudes.  For  future  demonstrations,  the  metric  of  fitted  polarizability  amplitude 
is  recommended  as  a  replacement  metric  as  these  values  are  invariant  to  array  position. 

4.3  OBJECTIVE:  REDUCTION  OF  FALSE  ALARMS 

This  is  the  primary  measure  of  the  effectiveness  of  this  technology.  By  collecting  high-quality, 
precisely  located  data,  we  expect  to  be  able  to  discriminate  munitions  from  scrap  and  frag  with 
high  efficiency.  This  metric  was  not  part  of  the  performance  criteria  for  the  former  Camp  SLO 
demonstration,  so  it  was  evaluated  only  for  the  APG  demonstration. 

4.3.1  Metric 

At  a  seeded  test  site  such  as  the  APG  standardized  test  site,  the  metric  for  false  alann  elimination 
is  straightforward.  We  prepared  a  ranked  dig  list  for  the  targets  we  interrogated  with  a  dig/no¬ 
dig  threshold  indicated,  and  Aberdeen  Test  Center  (ATC)  personnel  used  their  automated  scoring 
algorithms  to  assess  our  results. 

4.3.2  Data  Requirements 

The  identification  of  most  of  the  items  in  the  test  field  is  known  to  the  test  site  operators.  Our 
ranked  dig  list  was  the  input  for  this  objective,  and  ATC’s  standard  scoring  is  the  output. 

4.3.3  Success  Criteria 

The  objective  will  be  considered  to  be  met  if  more  than  50%  of  the  non- munitions  items  were 
labeled  as  no-dig  while  retaining  95%  of  the  munitions  items  on  the  dig  list. 
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4.3.4  Results 


This  objective  was  successfully  met.  The  TEMTADS  surveyed  anomalies  detected  by  the 
MTADS  magnetometer  system  in  the  blind  grid  and  indirect  fire  areas.  Efficiency  (E)  and  false 
positive  rejection  rate  (Rfp)  are  used  to  score  discrimination  performance  ability  at  two  specific 
operating  points  on  a  receiver  operating  characteristic  (ROC)  curve:  one  at  the  point  where  no 
decrease  in  probability  of  detection  (Pd)  is  incurred  and  the  other  at  the  operator-selected 
threshold.  Efficiency  is  defined  as  the  fraction  of  detected  ordnance  correctly  classified  as 
ordnance  and  the  false  positive  rejection  rate  is  defined  as  the  fraction  of  detected  clutter 
correctly  classified  as  clutter.  The  results  for  the  blind  grid  and  indirect  fire  test  areas  were  E  = 
0.99  and  Rfp  =  0.99,  and  E  =  0.98  and  Rfp  =  0.92  at  the  operating  point,  respectively.  With  no 
loss  of  Pd,  the  results  were  E  =  1.00  and  Rfp  =  0.95,  and  E  =  1.00  and  Rfp  =  0.58,  for  the  blind 
grid  and  the  indirect  fire  areas,  respectively.  These  data  are  summarized  from  Tables  7a  and  7c 
of  Reference  8. 

4.4  OBJECTIVE:  LOCATION  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  location  of  the  targets  marked  to  be  dug.  Large  location  errors  lead  to  confusion 
among  the  UXO  technicians  assigned  to  the  remediation  costing  time  and  often  lead  to  removal 
of  a  small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

4.4.1  Metric 

The  average  error  and  deviation  in  both  horizontal  axes  was  computed  for  the  items  which  are 
selected  for  excavation  during  the  validation  phase  of  the  demonstration.  We  provided  an 
estimated  position  for  all  targets  we  interrogated  to  the  appropriate  personnel  at  each  site,  and 
they  used  their  scoring  algorithms  to  assess  our  results.  At  APG,  all  the  items  were  emplaced 
and  the  locations  are  known.  Therefore  the  items  were  not  excavated.  Aggregate  results  for  the 
APG  demonstration  were  provided  by  ATC. 

4.4.2  Data  Requirements 

The  location  of  most  of  the  items  in  the  test  field  is  known  to  the  appropriate  personnel  at  each 
site.  Our  dig  list  was  the  input  for  this  objective,  and  a  standard  scoring  report  is  the  output. 

4.4.3  Success  Criteria 

The  objective  was  considered  to  be  met  if  the  average  position  error  (low  bias)  and  the  standard 
deviation  (accurate  location)  in  both  dimensions  was  less  than  10  and  15  cm,  respectively  for 
APG,  and  less  than  5  and  10  cm,  respectively  for  the  former  Camp  SLO. 

4.4.4  Results 

This  objective  was  not  successfully  met  in  full.  For  APG,  the  location  accuracy  of  fit  parameters 
generated  from  the  TEMTADS  array  data,  taken  from  Tables  9a  and  9c  of  Reference  8,  are 
within  5  cm  horizontally  (la)  and  6  cm  vertically  (la)  for  the  indirect  fire  area.  Horizontal 
errors  are  not  calculated  for  the  blind  grid.  The  vertical  accuracy  was  4  cm  (la)  for  the  blind 
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grid.  For  the  former  Camp  SLO,  the  average  northing  position  error  for  all  measured  data  was 

1.5  cm,  while  the  average  easting  position  error  was  3  cm.  The  standard  deviations,  however, 
were  larger  than  desired,  with  both  values  about  25  cm.  Excluding  those  anomalies  for  which 
multiple  targets  were  found  produces  negligible  improvement.  We  suspect  the  higher  values  are 
due  to  the  large  number  of  small,  low  SNR  clutter  items,  which  result  in  greater  uncertainty  in 
both  the  measured  and  fitted  values.  Indeed,  if  we  restrict  ourselves  to  those  anomalies  which  we 
classified  as  likely  UXO,  the  northing  and  easting  standard  deviations  drop  to  7  cm  and  5  cm, 
respectively. 

4.5  OBJECTIVE:  DEPTH  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  depth  of  the  targets  marked  to  be  dug.  Large  depth  errors  lead  to  confusion  among 
the  UXO  technicians  assigned  to  the  remediation  costing  time  and  often  leading  to  removal  of  a 
small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target. 

4.5.1  Metric 

The  standard  deviation  of  the  predicted  depths  with  respect  to  the  ground  truth  was  computed  for 
the  items  that  were  selected  for  excavation  during  the  validation  phase  of  the  fonner  Camp  SLO 
study.  At  APG,  all  the  items  were  emplaced  and  the  locations  are  known.  Therefore  the  items 
were  not  excavated.  Aggregate  results  for  the  APG  demonstration  were  provided  by  ATC. 

4.5.2  Data  Requirements 

The  anomaly  fit  parameters  and  the  ground  truth  for  the  excavated  items  were  required  to 
detennine  the  perfonnance  of  the  fitting  routines  in  terms  of  the  predicted  depth  accuracy. 

4.5.3  Success  Criteria 

This  objective  was  considered  as  met  if  the  average  error  in  depth  was  less  than  5  cm  and  the 
standard  deviation  less  than  10  cm. 

4.5.4  Results 

This  objective  was  not  met  successfully  in  full.  For  APG,  the  accuracy  of  the  depth  fit  parameter 
generated  from  the  TEMTADS  array  data,  taken  from  Tables  9a  and  9c  of  Reference  8,  was 
within  6  cm  vertically  (la)  for  the  indirect  fire  area.  The  vertical  accuracy  was  4  cm  (la)  for 
the  blind  grid.  For  the  SLO  demonstration,  the  average  depth  error  for  all  measured  data  was  2.5 
cm.  The  standard  deviation  was  a  bit  larger  than  desired,  with  a  value  of  14  cm.  Excluding  those 
anomalies  for  which  multiple  targets  were  found  and  restricting  the  analysis  to  those  anomalies 
classified  as  “likely  UXO”  to  exclude  low  SNR  clutter  items,  the  standard  deviation  reduces  to  7 
cm. 


4.6  OBJECTIVE:  PRODUCTION  RATE 

Even  if  the  perfonnance  of  the  technology  for  the  metrics  listed  above  is  satisfactory,  there  is  an 
economic  metric  to  consider.  There  is  a  known  cost  of  remediating  a  suspected  munitions  item. 
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If  the  cost  to  interrogate  a  target  is  greater  than  this  cost,  the  technology  will  be  useful  only  at 
sites  with  special  conditions  or  target  values.  Note,  however,  that  in  its  ultimate  implementation 
this  technology  will  result  in  reacquisition,  cued  interrogation,  and  target  flagging  in  one  visit  to 
the  site. 

4.6.1  Metric 

The  number  of  targets  interrogated  per  day  was  the  metric  for  this  objective.  Combined  with  the 
daily  operating  cost  of  the  technology,  this  gives  the  per-item  cost. 

4.6.2  Data  Requirements 

The  metric  was  determined  from  the  combination  of  available  field  logs  and  the  survey  results. 
The  field  logs  record  the  amount  of  time  per  day  spent  acquiring  the  data,  and  the  survey  results 
determine  the  number  of  anomalies  investigated  in  that  time  period. 

4.6.3  Success  Criteria 

For  the  APG  demonstration,  the  objective  was  considered  to  be  met  if  at  least  75  targets  were 
interrogated  each  survey  day.  For  the  former  Camp  SLO  demonstration,  the  production  rate 
target  was  125  anomalies/day. 

4.6.4  Results 

This  objective  was  successfully  met.  At  APG,  214  anomalies  were  investigated  on  the  blind  grid 
over  the  course  of  1.43  work  days,  or  on  average  149  anomalies/work  day,  and  694  anomalies 
were  investigated  in  the  indirect  fire  area  over  the  course  of  32  hours  and  30  minutes,  or  on 
average  170  anomalies/work  day.  At  the  former  Camp  SLO,  a  total  of  1547  anomalies 
(including  redos)  were  measured  over  a  10-day  period  for  an  average  of  155  anomalies/day. 

4.7  OBJECTIVE:  DATA  THROUGHPUT 

The  collection  of  a  complete,  high-quality  data  set  with  the  sensor  platform  is  critical  to  the 
downstream  success  of  any  UXO  classification  effort.  This  objective  considers  one  of  the  key 
data  quality  issues,  the  ability  of  the  data  analysis  workflow  to  support  the  data  collection  effort 
in  a  timely  fashion.  To  maximize  the  efficient  collection  of  high  quality  data,  a  series  of 
MTADS  standard  data  quality  checks  are  conducted  during  and  immediately  after  data  collection 
on  site.  Data  that  pass  the  QC  screen  are  then  processed  into  archival  data  stores.  Individual 
anomaly  analyses  are  then  conducted  on  those  archival  data  stores.  The  data  QC/preprocessing 
portion  of  the  workflow  needs  to  keep  pace  with  the  data  collection  effort  for  best  perfonnance. 

4.7.1  Metric 

The  throughput  of  the  data  quality  control  workflow  was  at  least  as  fast  as  the  data  collection 
process,  providing  real  time  feedback  to  the  data  collection  team  of  any  issues. 
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4.7.2  Data  Requirements 

The  data  analysts  log  books  provided  the  necessary  data  for  detennining  the  success  of  this 
metric. 


4.7.3  Success  Criteria 

This  objective  will  be  considered  met  if  all  collected  data  can  be  processed  through  the  data 
quality  control  portion  of  the  workflow  in  a  timely  fashion. 

4.7.4  Results 

This  objective  was  successfully  met.  Data  were  normally  downloaded  several  times  during  each 
workday,  and  quality  control  on  these  datasets  was  usually  completed  on  the  same  day.  QC 
checks  successfully  caught  missed  anomalies,  a  small  number  of  corrupt  data  files,  and  targets 
that  needed  remeasuring. 

For  low  SNR  targets,  our  standard  data  acquisition  parameters  may  not  be  sufficient.  The  system 
software  has  built  in  the  capability  to  vary  the  data  acquisition  parameters  on-the-fly  based  on 
flags  in  the  target  file  and  can  be  reconfigured  manually  as  required.  To  date  these  capabilities 
have  not  been  demonstrated  and  could  potentially  have  an  impact  on  data  throughput.  A  robust, 
consolidated  data  collection/data  QC  methodology  for  detennining  when  there  is  a  low  SNR 
anomaly  present  and  when  there  is  no  anomaly  present  is  necessary  to  accurately  and  efficiently 
utilize  these  capabilities.  This  issue  is  an  area  of  ongoing  research. 

4.8  OBJECTIVE:  ANALYSIS  TIME 

The  ultimate  implementation  of  this  technology  will  involve  on-the-fly  analysis  and 
classification.  The  time  for  this  will  be  limited  to  the  driving  time  to  the  next  anomaly  on  the 
list.  We  will  track  the  near-real-time  analysis  time  in  this  demonstration. 

4.8.1  Metric 

The  time  required  for  inversion  and  classification  per  anomaly  was  the  metric  for  this  objective 

4.8.2  Data  Requirements 

Analysis  time  was  determined  from  a  review  of  the  data  analysis  logs. 

4.8.3  Success  Criteria 

The  objective  was  considered  to  be  met  if  the  average  inversion  and  classification  time  was  less 
than  15  min. 

4.8.4  Results 

This  objective  was  successfully  met.  The  average  inversion  time  per  target  was  approximately 
2.5  min  on  our  field  laptop  computer.  The  total  average  analysis  time  amounted  to  12.5  min  per 
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anomaly.  For  these  extensive  tests  of  the  system  in  field  mode,  we  took  the  opportunity  to 
consider  various  discrimination  and  classification  methods,  some  of  which  proved  unfruitful.  As 
a  result  of  lessons  learned  from  this  undertaking,  we  expect  the  average  analysis  time  for  future 
field  runs  to  be  less. 

4.9  OBJECTIVE:  EASE  OF  USE 

This  qualitative  objective  is  intended  as  a  measure  of  the  long-term  usability  of  the  technology. 
If  the  operator  does  not  report  that  the  technology  is  easy  to  use,  shortcuts  that  can  compromise 
the  efficiency  of  the  technology  will  begin  to  creep  into  daily  operations. 

4.9.1  Data  Requirements 

This  objective  was  evaluated  based  on  operator  feedback. 

4.9.2  Results 

This  objective  was  successfully  met.  Based  on  vehicle  operator  feedback,  there  were  no 
significant  limitations  to  the  efficient  use  of  the  system  in  the  field.  Several  suggestions  were 
made  for  additional  improvements  to  the  navigation  and  data  collection  software.  They  have 
been  subsequently  incorporated. 
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5.0  SITE  DESCRIPTION 


Two  demonstrations  were  conducted  for  this  project.  The  first  was  conducted  at  the 
Standardized  UXO  Test  Site  located  at  the  APG,  MD,  May  through  June  2008.  The  second  was 
conducted  at  the  former  Camp  SLO,  CA,  ESTCP  UXO  Classification  Study  Demonstration  Site 
in  June  2009. 

5.1  APG  STANDARDIZED  UXO  TEST  SITE 

5.1.1  SITE  LOCATION  AND  HISTORY 

The  Standardized  UXO  Test  Site  is  adjacent  to  the  Trench  Warfare  facility  at  the  APG.  The 
specific  area  was  used  for  a  variety  of  ordnance  tests  over  the  years.  Initial  magnetometer  and 
EMI  surveys  conducted  by  the  MTADS  team  performed  after  a  “mag  and  flag”  survey  of  the 
same  area  identified  over  a  thousand  remaining  anomalies.  These  data  were  used  for  a  final 
cleanup  of  the  site  prior  to  the  emplacement  of  the  original  test  items.  Prior  to  the  two 
subsequent  reconfiguration  events,  unexplained  anomalies  identified  by  demonstrators  using  the 
site  were  also  investigated  and  removed. 

This  was  the  site  of  our  first  field  demonstration  of  this  combination  of  EMI  sensors  and  survey 
mode.  The  APG  site  is  located  close  to  our  base  of  operations  in  southern  Maryland  and 
therefore  minimizes  the  logistics  costs  of  the  deployment.  Use  of  this  site  allowed  us  to  receive 
validation  results  from  near-real-world  conditions  without  incurring  the  logistics  and  intrusive 
investigation  expenses  that  would  be  required  for  a  demonstration  at  a  live  site. 

5.1.2  SITE  TOPOGRAPHY  AND  GEOLOGY 

According  to  the  soils  survey  conducted  for  the  entire  area  of  APG  in  1998,  the  test  site  consists 
primarily  of  Elkton  Series  type  soil  [9].  The  Elkton  Series  consist  of  very  deep,  slowly 
permeable,  poorly  drained  soils.  These  soils  formed  in  silty  aeolin  sediments  and  the  underlying 
loamy  alluvial  and  marine  sediments.  They  are  on  upland  and  lowland  flats  and  in  depressions 
of  the  Mid-Atlantic  Coastal  Plain.  Slopes  range  from  0  to  2%. 

Overall,  the  demonstration  site  is  relatively  flat  and  level.  There  are  some  low-lying  areas  in  the 
northwest  portion  of  the  site  that  tend  to  have  standing  water  during  the  wet  periods  of  the  year. 
The  current  sensor  system  is  not  sufficiently  weatherproofed  to  operate  through  standing  water. 
However,  during  the  most  recent  reconfiguration,  the  areas  most  prone  to  being  underwater  were 
excluded  from  the  survey  scenarios.  Anomalies  that  were  located  underwater  or  near  water  at 
the  time  of  survey  were  deferred  until  the  end  of  the  survey  and  were  interrogated  by  carefully,  if 
less  efficiently,  maneuvering  the  array  into  position. 

5.1.3  MUNITIONS  CONTAMINATION 

The  area  currently  occupied  by  the  site  has  seen  an  extensive  history  of  munitions  use.  As  an 
example,  in  2003  we  conducted  a  magnetometer  survey  of  a  previously  unremediated  area 
directly  adjacent  to  the  site  [10],  In  a  survey  area  of  approximately  1  hectare,  we  identified  2479 
anomalies,  of  which  1921  were  amenable  to  a  model  fit  using  our  standard  analysis.  Historical 
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records  provided  by  ATC  and  previous  remediation  results  indicated  that  the  likely  munitions  of 
interest  for  this  site  were: 

•  Grenades,  Mkl,  Mkll,  and  French  VB  Rifle  without  chute 

•  Grenades,  French  VB  Rifle  with  chute 

•  60  mm  mortars  (including  2-inch  Smoke) 

•  3 -inch  Stokes  (Smoke  and  HE) 

•  105  mm  projectiles 

•  155  mm  proj  ectiles 

5.1.4  SITE  CONFIGURATION 

Figure  5  is  a  map  of  the  Standardized  UXO  Test  Site  at  APG.  The  calibration  and  blind  grids  are 
shown  along  with  the  various  open  field  areas. 

5.2  FORMER  CAMP  SAN  EUIS  OBISPO 

5.2.1  SITE  LOCATION  AND  HISTORY 

The  site  description  material  reproduced  here  taken  from  the  recent  site  investigation  (SI)  report 
[11].  More  details  can  be  obtained  in  the  report.  The  former  Camp  SLO  is  approximately  2101 
acres  situated  along  Highway  1,  approximately  5  miles  northwest  of  San  Luis  Obispo,  CA.  Most 
of  the  area  consists  of  mountains  and  canyons.  The  site  for  this  demonstration  is  a  mortar  target 
on  a  hilltop  in  MRS  05  (within  former  Rifle  Range  #12). 

Camp  SLO  was  established  in  1928  by  the  State  of  California  as  a  National  Guard  Camp. 
Identified  at  that  time  as  Camp  Merriam,  it  originally  consisted  of  5800  acres.  Additional  lands 
were  added  in  the  early  1940s  until  the  acreage  totaled  14,959.  From  1943  to  1946,  Camp  SLO 
was  used  by  the  U.S.  Anny  for  infantry  division  training  including  artillery,  small  arms,  mortar, 
rocket,  and  grenade  ranges.  Following  the  end  of  World  War  II,  a  small  portion  of  the  fonner 
camp  land  was  returned  to  its  former  private  owners.  The  U.S.  Anny  was  making  arrangements 
to  relinquish  the  rest  of  Camp  SLO  to  the  State  of  California  and  other  government  agencies 
when  the  conflict  in  Korea  started  in  1950.  The  camp  was  reactivated  at  that  time. 

The  U.S.  Anny  used  the  former  camp  during  the  Korean  War  from  1951  through  1953  where  the 
Southwest  Signal  Center  was  established  for  the  purpose  of  signal  corps  training.  Following  the 
Korean  War,  the  camp  was  maintained  in  inactive  status  until  it  was  relinquished  by  the  Army  in 
the  1960s  and  1970s.  Approximately  4685  acres  were  relinquished  to  the  General  Services 
Administration  (GSA)  in  1965.  GSA  then  transfened  the  property  to  other  agencies  and 
individuals  beginning  in  the  late  1960s  through  the  1980s;  most  was  transferred  for  educational 
purposes  (California  Polytechnic  State  University  and  Cuesta  College).  A  large  portion  of  Camp 
SLO  (the  original  5880  acres)  has  been  retained  by  the  California  National  Guard  (CNG)  and  is 
not  part  of  the  Formerly-Used  Defense  Site  (FUDS)  program. 


18 


19 


This  site  was  chosen  as  the  second  in  a  progression  of  increasingly  more  complex  sites  for 
demonstration  of  the  classification  process  as  part  of  the  ESTCP  UXO  Classification  Study.  The 
first  site  in  the  series,  former  Camp  Sibert,  had  only  one  target-of-interest  and  item  “size”  was  an 
effective  discriminant.  At  this  site,  there  are  at  least  four  targets-of-interest:  60-mm,  81 -mm,  and 
4.2-inch  mortars  and  2.36-inch  rockets.  This  introduces  another  layer  of  complexity  into  the 
process. 

5.2.2  SITE  TOPOGRAPHY  AND  GEOLOGY 

The  fonner  Camp  SLO  site  consists  mainly  of  mountains  and  canyons  classified  as  grassland, 
wooded  grassland,  woodland,  or  brush.  A  major  portion  of  the  site  is  identified  as  grassland  and 
is  used  primarily  for  grazing.  Los  Padres  National  Forest  (woodland)  is  located  to  the  north- 
northeastern  portion  of  the  site.  During  the  hot  and  dry  summer  and  fall  months,  the  intennittent 
areas  of  brush  occurring  throughout  the  site  become  a  critical  fire  hazard. 

The  underlying  bedrock  within  the  former  Camp  SLO  site  area  is  intensely  folded,  fractured,  and 
faulted.  The  site  is  underlain  by  a  mixture  of  metamorphic,  igneous,  and  sedimentary  rocks  less 
than  200  million  years  old.  Scattered  throughout  the  site  are  areas  of  fluvial  sediments  overlaying 
metamorphosed  material  known  as  Franciscan  melange.  These  areas  are  intruded  by  plugs  of 
volcanic  material  that  comprise  a  chain  of  former  volcanoes  extending  from  the  southwest 
portion  of  the  site  to  the  coast.  Due  to  its  proximity  to  the  tectonic  interaction  of  the  North 
American  and  Pacific  crustal  plates,  the  area  is  seismically  active.  Additional  details  are 
available  in  Reference  1 1 . 


5.2.3  MUNITIONS  CONTAMINATION 


A  large  variety  of  munitions  are  reported  to  have  been  used  at  the  former  Camp  SLO.  Munitions 
debris  from  the  following  sources  was  observed  throughout  MRS  05  during  the  2007  SI: 


•  4.2-inch  white  phosphorus  mortar 

•  4.2-inch  mortar  base  plate 

•  3. 5 -inch  rocket 

•  3  7-inm  projectile 

•  75 -mm  projectile 

•  flares  found  of  newer  metal, 
suspected  from  CNG  activities 


105 -mm  projectile 

60-mm  mortar 

81 -mm  mortar 

Practice  bomb 

30  caliber  casings  and  fuzes 


At  the  particular  site  of  this  demonstration,  60-mm,  81 -mm,  and  4.2-inch  mortars  and  mortar 
fragments  have  been  observed.  During  the  initial  EM61  Mkll  cart  survey,  two  2.36-inch  rockets 
were  found  on  the  surface.  The  excavation  of  two  50-ft  x  50-ft  grids  in  October  2008,  as  part  of 
the  preparatory  activities,  has  confirmed  these  observations  and  provided  infonnation  on  the 
depths  of  munitions  at  this  target  site. 

5.2.4  SITE  CONFIGURATION 


The  11.8-acre  demonstration  site  is  shown  in  Figure  6  as  a  series  of  30-m  x  30-m  cells  with  a 
topographical  map  as  the  background.  The  cells  are  color-coded  based  on  the  data  collection 
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systems  that  were  deployed  on  them,  tan  color  for  all  systems  and  blue  for  vehicular  systems 
only.  The  site  spans  a  significant  fraction  of  the  hillside  that  is  the  historical  mortar  target.  The 
test  pit  was  located  near  the  logistics  base,  and  the  calibration  strip  was  located  outside  the  inner 
fence  line,  convenient  to  the  site  access  road. 
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Figure  6.  ESTCP  UXO  Classification  Study  demonstration  site 
at  the  former  Camp  San  Luis  Obispo. 

The  site  is  shown  as  a  series  of  30-m  x  30-m  cells.  See  the  text  for  further  discussion. 
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6.0  TEST  DESIGN 


6.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

Each  demonstration  was  designed  to  be  executed  in  two  stages.  The  first  stage  consisted  of  a 
standard  MTADS  dynamic  survey  of  the  site.  For  the  APG  demonstration,  the  MTADS 
magnetometer  array  was  the  survey  instrument.  The  details  of  the  magnetometer  survey  can  be 
found  in  Reference  5.  For  the  former  Camp  SFO  demonstration,  the  MTADS  EM61  Mkll  array 
was  the  survey  instrument.  The  details  of  the  Mkll  survey  can  be  found  in  Reference  12.  The 
choice  of  the  appropriate  technology  for  the  first  stage  of  the  survey  is  governed  by  the 
combination  of  site  history  (expected  target  of  interests  [TOI]  and  clutter)  and  opportunity  costs. 
The  APG  Standardized  Test  Site  had  recently  completed  a  reconfiguration  in  April  2008,  and 
NRF  was  requested  to  conduct  a  magnetometer  survey  of  the  entire  APG  Standardized  UXO 
Test  Site.  Past  experience  at  APG  and  with  the  seeded  TOIs  indicated  that  the  magnetometer 
survey  would  provide  acceptable  detection  for  the  later  TEMTADS  survey.  For  the  fonner 
Camp  SLO  demonstration,  both  the  MTADS  magnetometer  and  EM61  Mkll  array  were 
deployed  to  the  site  and  available.  Based  on  characterization  measurements  of  both  the  site 
geology  and  the  TOIs  with  both  arrays,  the  EM61  Mkll  array  data  had  a  significantly  higher 
SNR  for  detection  of  the  known  TOIs,  so  it  was  selected. 

Anomaly  locations  were  identified  from  the  survey  data  in  a  combination  automated/manual 
method.  A  data  segment  around  each  anomaly  center  was  extracted  and  analyzed  using  the  UX- 
Analyze  subsystem  of  the  Oasis  montaj  software  package  to  fit  the  data  to  a  dipole  model  and 
extract  the  associated  fit  parameters  (position,  depth,  equivalent  size).  These  fit  results 
constituted  the  source  anomaly  list  for  the  second  stage  of  each  demonstration. 

This  method  relies  on  the  establishment  of  an  anomaly  detection  threshold.  At  the  fonner  Camp 
Sibert  demonstration  site,  a  single  munitions  type  was  present  [13].  Pit  measurements  at  various 
depths  and  orientations  of  an  example  article  were  made  and  bounding  response  curves 
generated  for  the  4.2-inch  mortar,  the  munitions  of  interest.  The  anomaly  detection  threshold 
was  then  set  based  on  the  least  favorably  predicted  response  at  the  U.S.  Corps  of  Engineers 
(USCOE)  standard  1 1  times  depth.  These  demonstration  sites  each  contained  different  mixes  of 
emplaced  munitions  and  suspected  existing  munitions  contamination.  Individual  anomaly 
detection  thresholds  were  established  for  each  site/area  based  on  sets  of  pit  measurements  made 
for  each  of  the  emplaced  items.  For  each  site/area,  the  smallest  appropriate  least  favorable 
response  was  used  to  determine  the  threshold.  The  details  of  the  anomaly  selection  process, 
including  the  response  curves,  can  be  found  in  the  demonstration  data  reports  [5,  12]. 

The  second  stage  of  each  demonstration  was  the  survey  of  each  site/area  using  the  TEMTADS 
array  developed  as  part  of  ESTCP  Project  MR-0601.  The  array  was  positioned  roughly  over  the 
center  of  each  anomaly  on  the  source  anomaly  list  and  a  data  set  collected.  Each  data  set  was 
then  inverted  using  the  data  analysis  methodology  discussed  in  Section  7.3,  estimated  target 
parameters  determined,  and  ultimately  a  classification  made  for  each  anomaly.  The  resulting 
prioritized  dig  lists  were  then  submitted  to  either  the  ATC  or  Institute  for  Defense  Analyses 
(IDA)  for  scoring  and  performance  assessment. 

The  schedule  of  field  testing  activities  is  provided  in  Figure  7  as  a  Gantt  chart. 
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Figure  7.  Schedule  of  field  testing  activities. 


6.2  SITE  PREPARATION 

Each  demonstration  site  had  been  previously  configured  with  clearly  marked  calibration  and 
open  field  scenarios.  At  least  one  GPS  control  point  was  provided  at  each  site.  Basic  facilities 
such  as  portable  toilets  and  field  buildings  were  provided  at  APG  and  acquired  for  SLO.  Secure 
storage  for  larger  vehicles  and  sensor  arrays  was  limited  at  both  sites.  A  40-ft  shipping  container 
was  mobilized  to  each  site  for  the  duration  of  each  demonstration  to  provide  convenient,  secure 
storage  for  the  MTADS  tow  vehicle  and  the  sensor  trailer.  The  container  was  removed  at  the  end 
of  each  demonstration. 

6.3  SYSTEMS  SPECIFICATION 

This  demonstration  was  conducted  using  the  NRL  MTADS  tow  vehicle  and  subsystems.  The 
tow  vehicle  and  each  subsystem  are  described  further  in  the  following  sections. 

6.3.1  MTADS  Tow  Vehicle 

The  MTADS  has  been  developed  by  the  NRL  Chemistry  Division  with  support  from  ESTCP. 
The  MTADS  hardware  consists  of  a  low-magnetic-signature  vehicle  that  is  used  to  tow  the 
different  sensor  arrays  over  large  areas  (10-25  acres/day)  to  detect  buried  UXO.  The  MTADS 
tow  vehicle  and  TEMTADS  array  are  shown  in  Figure  8. 


Figure  8.  MTADS  tow  vehicle  and  TEMTADS  array. 
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6.3.2  RTK  GPS  System 


Positioning  is  provided  using  cm-level  real-time  kinematic  (RTK)  GPS  receivers.  To  achieve 
cm-level  precision,  a  fixed  reference  base  station  is  placed  on  an  established  first-order  survey 
control  point  near  the  survey  area.  The  base  station  transmits  corrections  to  the  GPS  rover  at  1 
Hertz  (Hz)  via  a  radio  link. 

The  TEMTADS  array  is  located  in  three-dimensional  space  using  a  three-receiver  RTK  GPS 
system  shown  schematically  in  Figure  3  [14].  The  three-receiver  configuration  extends  the 
concept  of  RTK  operations  from  that  of  a  fixed  base  station  and  a  moving  rover  to  moving  base 
stations  and  moving  rovers.  The  lead  GPS  antenna  (and  receiver,  Main)  receives  corrections 
from  the  fixed  base  station  at  1  Hz.  The  corrected  position  of  the  Main  GPS  antenna  is  reported 
at  10-20  Hz.  The  Main  receiver  also  operates  as  a  moving  base,  transmitting  corrections  (by 
serial  cable)  to  the  next  GPS  receiver  (AVR1),  which  uses  the  corrections  to  operate  in  RTK 
mode. 

A  vector  (AVR1,  heading  [yaw],  angle  [pitch],  and  range)  between  the  two  antennae  is  reported 
at  10  Hz.  AVR1  also  provides  moving  base  corrections  to  the  third  GPS  antenna  (AVR2),  and  a 
second  vector  (AVR2)  is  reported  at  10  Hz.  All  GPS  measurements  are  recorded  at  full  RTK 
precision,  ~2-5  cm.  The  GPS  position  is  averaged  for  2  seconds  (s)  as  part  of  the  data 
acquisition  cycle.  The  averaged  position  and  orientation  infonnation  are  then  recorded  to  the 
position  data  file. 

6.3.3  Time-Domain  Electromagnetic  Sensor 

The  TEMTADS  array  is  a  5x5  square  array  of  individual  sensors.  Each  sensor  has  dimensions 
of  40  cmx40  cm,  for  an  array  of  2  m  x  2  m  overall  dimensions.  The  rationale  of  this  array 
design  is  discussed  in  Reference  15.  The  result  is  a  cross-track  and  down-track  separation  of  40 
cm.  Sensor  numbering  is  indicated  in  Figure  3.  The  transmitter  electronics  and  the  data 
acquisition  computer  are  mounted  in  the  tow  vehicle.  Custom  software  written  by  NRL  provides 
both  navigation  to  the  individual  anomalies  and  data  acquisition  functionality.  After  the  array  is 
positioned  roughly  centered  over  each  anomaly,  the  data  acquisition  cycle  is  initiated.  Each 
transmitter  is  fired  in  a  sequence  winding  outward  from  the  center  position  (12)  in  a  clockwise 
direction.  The  received  signal  is  recorded  for  all  25  Rx  coils  for  each  transmit  cycle.  The 
transmit  pulse  waveform  duration  is  2.7  s.  While  it  is  possible  to  record  the  entire  decay 
transient  at  500  MHz,  we  have  found  that  binning  the  data  into  115  time  gates  simplifies  the 
analysis  and  provides  additional  signal  averaging  without  significant  loss  of  temporal  resolution 
in  the  transient  decays  [16].  The  data  are  recorded  in  a  binary  format  as  a  single  file  with  25  data 
points  (one  data  point  per  Tx  cycle). 

6.4  DATA  COLLECTION 

6.4.1  Scale  of  Demonstration 

The  APG  demonstration  was  conducted  at  the  APG  Standardized  UXO  Test  Site.  A 
magnetometer  survey  was  conducted  on  the  calibration  and  blind  grids,  as  well  as  the  indirect 
fire  area  (approximately  4.3  acres)  on  May  7,  2008.  The  TEMTADS  array  surveyed  the 
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calibration  and  blind  grids.  The  array  was  also  deployed  to  approximately  700  anomalies  in  the 
indirect  fire  area  that  were  detected  from  the  magnetometer  data  set.  The  TEMTADS  array 
portion  of  the  demonstration  occurred  June  16-23,  2008. 

A  cued  discrimination  survey  of  the  fonner  Camp  SLO  demonstration  site  was  conducted  within 
the  11.8-acre  final  demonstration  site  of  approximately  1500  previously  identified  anomalies 
from  the  anomaly  list  generated  from  the  MTADS  EM61  Mkll  data  set.  This  survey  was 
conducted  using  the  NRL  TEMTADS.  The  MTADS  EM61  Midi  data  collection  occurred  May 
11  -  18,  2009.  The  TEMTADS  demonstration  occurred  June  8-18,  2009. 

6.4.2  Sample  Density 

Magnetometer  data  were  collected  with  nominal  down-track  spacing  of  6  cm  and  cross-track 
spacing  of  25  cm.  EM61  Midi  EMI  data  were  collected  with  a  nominal  down-track  spacing  of 
15  cm  and  a  cross-track  spacing  of  50  cm.  Two  orthogonal  surveys  were  conducted  to  increase 
target  illumination  and  data  density.  The  EMI  data  spacing  for  the  TEMTADS  is  fixed  at  40  cm 
in  both  directions  by  the  array  design. 

6.4.3  Quality  Checks 

Since  the  TEMTADS  operates  in  a  cued  mode,  the  data  QC  procedures  and  checks  differ  from 
that  of  survey  mode  instruments.  The  status  of  the  RTK  GPS  system  can  be  visually  verified  by 
the  operator  prior  to  starting  the  data  collection  cycle,  assuring  that  the  position  and  orientation 
information  are  valid  (fix  quality  [FQ]  3,  Position  Dilution  of  Precision  [PDOP]  <  4)  during  the 
collection  period. 

Two  data  quality  checks  were  performed  on  the  TEMTADS  data.  After  background  subtraction, 
contour  plots  of  the  signal  were  generated  for  the  25  transmit/receive  pairs  at  a  decay  time  of 
0.042  ms.  An  example  of  a  good  data  set  from  a  single  anomaly  with  a  large  SNR  is  shown  in 
Figure  9  for  the  APG  Calibration  Area  item  16.  The  plots  were  visually  inspected  to  verify  that 
there  was  a  well-defined  anomaly  without  extraneous  signals  or  dropouts.  QC  on  the 
transmit/receive  cross  terms  was  based  on  the  dipole  inversion  results.  Our  experience  has 
shown  that  data  glitches  show  up  as  reduced  dipole  fit  coherence. 


Figure  9.  Monostatic  QC  contour  plot  for  Calibration  Area  item  16. 
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The  issue  of  when  multiple  objects  are  found  to  be  under  the  array  simultaneously,  generating 
overlapping  signatures,  can  also  be  addressed  at  this  point  in  the  data  QC  process.  An  example 
case  is  shown  in  Figure  10.  There  are  two  apparent  issues  in  the  data  set.  First,  there  appears  to 
be  a  small,  shallow  bit  of  scrap  on  top  of  the  target.  Second,  there  was  a  bit  of  scrap  present  in 
the  background  file  used.  This  latter  issue  is  seen  in  the  data  from  element  0.  The  fit  coherence 
using  all  elements  is  0.652.  If  elements  0,  5,  10,  11,  15,  16,  17,  20,  21,  22  are  excluded,  the  fit 
coherence  for  the  remaining  elements  is  0.985  and  the  betas  match  our  library  105  mm  FIEAT 
betas  very  well. 
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Figure  10.  Monostatic  QC  contour  plot  for  example  anomaly. 

Any  data  set  that  was  deemed  unsatisfactory  by  the  data  analyst  was  flagged  and  not  processed 
further.  The  anomaly  corresponding  to  the  flagged  data  was  logged  for  future  re-acquisition. 
Data  which  meet  these  standards  are  of  the  quality  typical  of  the  MTADS  system. 

6.4.4  Data  Summary 

The  primary  performance  metrics  for  this  demonstration  are  the  classification  performance 
results  for  the  TEMTADS  array  at  the  two  demonstration  sites.  The  first  demonstration  was 
conducted  at  the  APG  Standardized  UXO  Test  Site.  The  performance  results  are  provided  by  the 
site  managers  after  the  classification  rankings  are  submitted  [8].  The  ground  truth  of  this  site  is 
held  by  the  principal  investigators  (PI),  and  results  are  only  reported  in  aggregate.  For  the 
former  Camp  SLO  demonstration,  there  is  no  single  set  of  performance  results  as  the  UXO 
Classification  Study  involves  multiple  analysis  of  each  collected  data  set.  See  the  UXO 
Classification  Study  for  former  Camp  SLO  Final  Report  [17]  for  results  by  combination. 

6.5  VALIDATION 

6.5.1  Aberdeen  Proving  Grounds 

With  the  exception  of  the  Calibration  Grid,  the  ground  truth  for  the  standardized  sites  is  held 
back  from  individual  technology  demonstrators  to  preserve  the  utility  of  the  blind  grid  and  open 
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field  areas.  Results  from  the  blind  grid  and  the  indirect  fire  area  were  submitted  to  ATC  for 
perfonnance  evaluation.  Scoring  results  have  been  received  and  are  available  [8]. 

6.5.2  Former  Camp  San  Luis  Obispo 


At  the  conclusion  of  data  collection  activities,  all  anomalies  on  the  master  anomaly  list 
assembled  by  the  Program  Office  were  excavated.  Each  item  encountered  was  identified, 
photographed,  its  depth  measured,  its  location  detennined  using  cm-level  GPS,  and  the  item 
removed  if  possible.  All  nonhazardous  items  were  saved  for  later  in-air  measurements  as 
appropriate.  This  ground  truth  infonnation,  once  released,  was  used  to  validate  the  objectives 
listed  in  Section  4.0. 
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7.0  DATA  ANALYSIS  AND  PRODUCTS 


The  TEMTADS  is  a  cued  UXO  classification  system.  There  were  two  parts  to  the  data 
collection  in  each  demonstration:  an  initial  survey,  which  served  primarily  to  locate  targets,  and 
TEMTADS  measurements  made  over  each  target  detected  in  the  survey.  The  TEMTADS  data 
were  used  for  target  classification. 

7.1  PREPROCESSING 

The  collected  survey  data  were  preprocessed  on  site  for  quality  assurance  purposes  using 
standard  TEMTADS  procedures  and  checks  as  discussed  in  Section  6.4.3  and  References  5  and 
6. 

The  TEMTADS  has  25  transmitters/receiver  pairs.  For  each  transmit  pulse,  the  response 
measured  at  all  of  the  receivers  was  recorded  simultaneously.  For  each  target,  a  25x25  xN  data 
array  is  generated,  where  N  is  the  number  of  recorded  time  gates.  The  current  system 
configuration  bins  the  data  in  121  logarithmically  spaced  gates.  During  the  preprocessing  step, 
the  recorded  signals  are  normalized  by  the  transmitter  currents  to  account  for  any  transmitter 
variations.  To  account  for  time  delay  due  to  effects  of  the  receive  coil  and  electronics,  we 
subtract  0.028  ms  from  the  nominal  gate  times  [18].  The  delay  was  previously  determined 
empirically  by  comparing  measured  responses  for  test  spheres  with  theory.  The  measured 
responses  include  distortions  due  to  transmitter  ringing  and  related  artifacts  out  to  about  40  ps. 
Consequently  we  included  only  the  responses  from  decay  times  beyond  40  ps  in  our  analyses. 
This  leaves  115  gates  spaced  logarithmically  between  0.042  ms  and  25  ms. 

The  background  response  was  subtracted  from  each  target  measurement  using  data  collected  in  a 
nearby  target-free  region,  or  background.  Background  locations  were  selected  from  quiet  areas 
observed  in  the  survey  data.  All  the  background  measurements  were  intercompared  to  evaluate 
background  variability  and  to  identify  outlier  measurements  potentially  corresponded  to 
measurements  over  nonferrous  targets.  We  have  not  observed  significant  background  variability 
in  our  measurements  at  our  Blossom  Point  test  site  and  were  able  to  use  blank  ground 
measurements  from  100  m  away  for  background  subtraction  on  targets  in  the  test  field. 

Geo-referencing  of  the  array  data  is  based  on  the  GPS  data,  which  gives  the  location  of  the  center 
of  the  array  and  the  orientation  of  the  array.  Sensor  locations  within  the  array  are  fixed  by  the 
array  geometry.  Dipole  inversion  of  the  array  data  (Section  7.3)  detennines  target  location  in 
local  array-based  coordinates.  The  local  position  was  then  transformed  to  absolute  coordinates 
using  the  array  location  and  orientation  detennined  from  the  corresponding  GPS  data. 

7.2  TARGET  SELECTION  FOR  DETECTION 

Targets  were  selected  from  the  survey  data  using  the  threshold  exceedance  methods  described  in 
References  5  and  12  using  a  physics-based  threshold  in  Oasis  montaj.  The  data  chips  associated 
with  the  detected  anomalies  were  then  processed  in  an  automatic  processing  mode  of  UX- 
Analyze.  The  results  of  the  automatic  processing  were  then  reviewed  in  UX-Analyze’s 
interactive  mode  to  allow  operator  experience  to  be  included  in  the  target  selection.  In  the  case 
of  the  APG  demonstration,  a  small  number  of  additional  anomalies  were  identified  by  the 


operator  and  added  to  the  anomaly  list.  For  the  fonner  Camp  SLO  demonstration,  the  operator 
coalesced  double-peaked  features  into  a  single  anomaly  where  appropriate. 

7.3  PARAMETER  ESTIMATION 

The  raw  signature  data  from  the  TEMTADS  reflect  details  of  the  sensor/target  geometry  as  well 
as  inherent  EMI  response  characteristics  of  the  targets  themselves.  In  order  to  separate  out  the 
intrinsic  target  response  properties  from  sensor/target  geometry  effects,  we  invert  the  signature 
data  to  estimate  principal  axis  magnetic  polarizabilities  for  the  targets.  The  TEM  data  are 
inverted  using  the  standard  induced  dipole  response  model  wherein  the  effect  of  eddy  currents 
set  up  in  the  target  by  the  primary  field  is  represented  by  a  set  of  three  orthogonal  magnetic 
dipoles  at  the  target  location  [19]. 

Given  a  set  of  measurements  of  the  target  response  with  varying  geometries  or  “look  angles”  at 
the  target,  the  data  can  be  inverted  to  determine  the  (X,  Y,  Z)  location  of  the  target,  the 
orientation  of  its  principal  axes  ((j),  0,  i[r),  and  the  principal  axis  polarizabilities  (Pi,  P2,  P3).  The 
basic  idea  is  to  search  out  the  set  of  nine  parameters  (X,  Y,  Z,  (J),  0,  i[r,  Pi,  P2,  P3)  that  minimizes 
the  difference  between  the  measured  responses  and  those  calculated  using  the  dipole  response 
model. 

For  the  TEMTADS  data,  inversion  is  accomplished  by  a  two-stage  method.  In  the  first  stage,  the 
target’s  (X,  Y,  Z)  dipole  location  is  solved  for  non-linearly.  At  each  iteration  within  this 
inversion,  the  nine  element  polarizability  tensor  (B)  is  solved  linearly.  The  non-linear  inversion 
is  done  simultaneously  over  all  time  gates,  such  that  the  dipole  (X,  Y,  Z)  location  applies  to  all 
decay  times.  At  each  time  gate,  the  eigenvalues  and  angles  are  extracted  from  the  polarizability 
tensor.  In  the  second  stage,  six  parameters  are  used:  the  three  spatial  parameters  (X,  Y  ,Z)  and 
three  angles  representing  the  yaw,  pitch,  and  roll  of  the  target  (Euler  angles  (J),  0,  i[r).  In  this 
second  stage  both  the  target  location  and  its  orientation  are  required  to  remain  constant  over  all 
time  gates.  The  value  of  the  best  fit  X,  Y,  and  Z  from  the  first  stage,  and  the  median  value  of  the 
first-stage  angles  are  used  as  an  initial  guess  for  this  stage.  Additional  loops  over  depth  and 
angles  are  included  to  better  ensure  finding  the  global  minimum. 

Not  every  target  from  the  detection  surveys  had  a  strong  enough  TEM  response  to  support 
extraction  of  target  polarizabilities.  All  the  data  were  run  through  the  inversion  routines,  and  the 
results  were  manually  screened  to  identify  those  targets  that  could  not  be  reliably  classified. 
Several  criteria  were  used  in  this  process:  signal  strength  relative  to  background,  dipole  fit  error 
(difference  between  data  and  model  fit  to  data),  the  visual  appearance  of  the  polarizability 
curves. 

For  moderate-to-good  SNR,  our  derived  polarizabilities  can  be  directly  compared  with  those  of 
our  target  library.  In  these  cases,  a  metric  was  computed  based  on  how  well  the  amplitude  and 
the  ratio  of  the  target  polarizabilities  match  those  of  library  objects.  For  anomalies  in  the  APG 
demonstration  with  lower  SNR  or  for  cases  where  a  visual  inspection  suggested  a  match,  but  the 
match  quality  was  below  our  cutoff,  target  classifications  were  done  using  library  matching 
procedures  similar  to  those  used  by  Sky  Research,  Lawrence  Berkeley  Lab,  and  ourselves  in  the 
ESTCP  Discrimination  Study  Pilot  Program  at  the  former  Camp  Sibert  [20].  The  fit  quality  of 
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an  unconstrained  dipole  inversion  of  the  TEMTADS  data  was  compared  to  the  fit  quality  of  a 
dipole  fit  constrained  by  a  set  of  principal  axis  polarizabilities  drawn  from  a  signature  library. 

The  scoring  rules  for  the  SLO  demonstration  were  different  than  those  used  for  APG 
submissions,  including  the  introduction  of  the  “can’t  decide”  category;  therefore,  a  similar  but 
modified  process  was  used.  The  primary  library  match  algorithm  compares  the  polarizabilities 
of  an  unknown  target  with  each  library  entry  based  on  three  criteria:  the  amplitude  of  (3 1 ,  and  the 
two  shape  parameters,  P2/P1  and  (B3/P1.  For  each  anomaly,  this  match  was  performed  on  both  the 
polarizabilities  from  the  single  dipole  fit  and  those  of  each  target  obtained  by  a  multi-dipole  fit. 
The  metric  was  set  to  whichever  set  of  P’s  produces  the  best  match. 

7.4  CLASSIFIER  AND  TRAINING 

Prior  to  the  demonstrations  we  collected  training  data  in  air  for  all  of  the  14  standard  APG 
ordnance  targets  and  the  four  known  munitions  types  for  the  fonner  Camp  SLO  site.  These  data 
were  used  to  generate  the  fit  library  entries.  Many  of  the  targets  are  composites  of  two  or  more 
distinct  parts,  like  a  steel  body  combined  with  an  aluminum  tail  assembly.  Depending  on  the 
distance  between  the  sensors  and  the  target,  such  items  can  exhibit  a  range  of  slightly  different 
EMI  signatures  corresponding  to  excitation  from  different  directions.  We  included 
measurements  with  the  target  oriented  nose  up,  towards  the  sensor  array,  nose  down,  away  from 
the  array,  flat  and  obliquely  in  the  fit  library.  We  have  assembled  a  fairly  extensive 
polarizability  database  for  clutter  items  recovered  from  several  different  sites.  This  library  was 
used  as  training  data  for  establishing  UX O/clutter  discrimination  boundaries  on  the  direct  match 
metric  and  on  the  coherence  ratio. 

7.5  DATA  PRODUCTS 

The  primary  data  products  generated  are  the  ranked  dig  lists  submitted  for  scoring.  Additionally, 
the  archival  data  sets,  stored  individually  by  site  and  by  area  are  provided  for  future  data 
analysis/classification  efforts. 
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8.0  PERFORMANCE  ASSESSMENT 


The  TEMTADS  array  was  constructed  in  2007,  field  tested  at  the  APG  Standardized  UXO  Test 
Site  in  June  2008  [5,  8],  and  participated  in  the  ESTCP  UXO  Classification  Study  at  the  fonner 
Camp  SLO  site  in  June  2009  [6],  Due  to  the  nature  of  the  demonstrations  at  each  site  and  the 
employed  scoring  methods,  separate  performance  objectives  were  used  for  each  demonstration. 
In  this  section,  as  in  Section  4.0,  the  perfonnance  results  are  discussed  as  a  whole. 


The  perfonnance  objectives  for  these  demonstrations  were  summarized  in  Table  1.  The  analyses 
which  were  used  to  evaluate  these  criteria  are  discussed  in  the  following  sections.  The  ATC 
scoring  report  for  the  demonstration,  Reference  8,  provides  the  source  material  for  evaluating 
several  of  the  perfonnance  objectives. 

8.1  OBJECTIVE:  CALIBRATION  STRIP  RESULTS 

This  objective  is  supported  by  each  sensor  system  operating  in  good  working  order  and 
collecting  physically  valid  data  each  day.  A  calibration  strip,  consisting  of  two  samples  of  each 
of  the  four  targets  of  interest  plus  two  shotputs  (Table  2),  was  emplaced  on  site  as  a  means  of 
verifying  proper  system  operation  on  a  daily  basis.  The  strip  was  surveyed  twice  daily,  once  at 
the  beginning  of  the  day,  and  once  at  the  end. 

Table  2.  Details  of  former  Camp  SLO  Calibration  Strip. 


Item 

ID 

Easting 

(m) 

Northing 

(m) 

Depth 

(m) 

Inclination 

Azimuth 
(°cw  from  N) 

shotput 

705,417.00 

3,913,682.00 

0.25 

N/A 

N/A 

81  mm 

705,420.92 

3,913,687.63 

0.30 

Vertical  down 

0 

81  mm 

705,424.10 

3,913,692.95 

0.30 

Horizontal 

120 

60  mm 

705,427.53 

3,913,698.54 

0.30 

Vertical  down 

0 

60  mm 

705,430.85 

3,913,704.10 

0.30 

Horizontal 

120 

4.2-inch  mortar 

705,434.54 

3,913,709.44 

0.30 

Vertical  down 

0 

4.2-inch  mortar 

705,437.99 

3,913,715.04 

0.30 

Horizontal 

120 

2.36-inch  rocket 

705,441.46 

3,913,720.24 

0.30 

Vertical  down 

0 

2.36-inch  rocket 

705,445.00 

3,913,725.91 

0.30 

Horizontal 

120 

shotput 

705,448.50 

3,913,731.50 

0.35 

N/A 

N/A 

Prior  to  our  demonstration  at  the  former  Camp  SLO,  we  obtained  measurements  of  each  of  the 
four  munitions  types  with  the  TEMTADS  array.  Using  the  derived  polarizabilities  from  these 
measurements  and  the  forward  model  of  our  TEMTADS  code,  response  curves  were  generated 
for  each  munition.  These  curves  plot  the  minimum  expected  peak  signal  (the  peak  signal  when 
the  target  is  oriented  horizontally)  and  the  maximum  expected  peak  (the  peak  signal  when  the 
target  is  oriented  horizontally)  from  each  target  as  a  function  of  distance  below  sensor  or  depth 
below  ground.  The  peak  signal  is  that  obtained  from  the  monostatic  tenn  of  the  center  element 
of  the  array  at  the  first  time  gate.  The  measured  peak  signals  for  the  60-nnn  mortars  are  given  in 
Figure  1 1  as  an  example. 
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This  objective  was  not  successfully  met  in  full.  The  maximum  (red)  and  minimum  (blue) 
response  curves  are  plotted.  For  self-consistency,  we  have  plotted  the  measurements  at  the  mean 
inverted  depth  rather  than  the  reported  depths.  The  measured  values  generally  fit  well  within  the 
bounding  curves,  with  the  4.2-inch  mortar  and  shotput  results  the  poorest,  with  a  tendency  to 
underestimate  the  peak  value. 

Careful  examination  of  the  data  shows  that  this  variation  is  the  result  of  the  shot-to-shot 
precision  with  which  the  array  can  be  positioned  in  exactly  the  same  spot  each  time.  Because  the 
response  curves  are  generated  assuming  the  target  is  directly  below  the  sensor,  any  offset  in  the 
sensor  position  will  result  in  the  derived  peak  signal  being  smaller  than  that  predicted  by  the 
curve,  as  is  observed.  For  future  demonstrations,  the  metric  of  polarizability  amplitude  is 
recommended  as  these  values  are  invariant  to  array  position. 

Table  3  shows  the  mean  and  standard  deviation  in  the  peak  measured  signal  for  all  the  emplaced 
Calibration  Strip  items.  Of  the  10  items,  three  give  RMS  variations  above  our  stated  goal  of 
15%,  with  only  T-008  being  significantly  above.  This  target  consistently  inverted  to  a  depth  of 
18.5  cm,  and  thus  showed  the  largest  spatial  variation  in  signal.  This,  coupled  with  the  array  not 
always  being  properly  centered  on  the  target,  explains  the  larger  variation  for  this  object. 
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Table  3.  Peak  signals  for  former  Camp  SLO  calibration  strip  emplaced  items. 


Item  ID 

Description 

Depth 

(m) 

Mean  Signal 
(mV/Amp) 

Std  Dev. 
(mV/Amp, Ict) 

Variation 

(%) 

shotput 

0.25 

27.44 

4.18 

15.23 

81  mm 

0.30 

15.46 

1.39 

8.99 

81  mm 

0.30 

10.96 

0.91 

8.30 

60  mm 

0.30 

8.74 

0.89 

10.18 

60  mm 

0.30 

6.73 

1.13 

16.79 

4.2-inch  mortar 

0.30 

52.38 

5.82 

11.11 

4.2-inch  mortar 

0.30 

41.75 

5.25 

12.57 

2.36-inch  rocket 

0.30 

32.70 

7.91 

24.19 

2.36-inch  rocket 

0.30 

4.04 

0.32 

7.92 

shotput 

0.35 

11.82 

1.52 

12.86 

The  variability  and  accuracy  of  the  positional  fit  parameters  for  the  Calibration  Strip  emplaced 
items  were  determined  by  comparing  the  inverted  northing  and  easting  values  with  reported 
values.  These  numbers  are  shown  in  Table  4.  We  give  the  mean  vector  offset  dR  between  the 
inverted  and  reported  position  as  well  as  the  easting  (dx)  and  northing  (dy)  components.  The  dx 
and  dy  values  are  computed  using  the  inverted  positions  minus  the  reported  ones. 

Table  4.  Position  accuracy  and  variability  for  former  Camp  SLO  calibration  strip 

emplaced  items. 


Item 

ID 

Description 

Depth 

(m) 

Mean 

dR 

(m) 

Std  Dev 
dR 

(m,  Ict) 

Mean 

dx 

(m) 

Std  Dev 
dx 
(m) 

Mean 

dy 

(m) 

Std  Dev 
dy 

(m) 

shotput 

0.25 

0.209 

0.007 

0.167 

0.013 

0.124 

o.on  ; 

81  mm 

0.30 

0.413 

0.008 

-0.372 

0.008 

-0.178 

0.010 

81  mm 

0.30 

0.058 

0.014 

0.028 

0.010 

-0.049 

60  mm 

0.30 

0.026 

0.013 

0.013 

0.010 

0.015 

60  mm 

0.30 

0.038 

0.012 

-0.003 

0.011 

0.035 

4.2-inch  mortar 

0.30 

0.098 

0.008 

0.082 

0.009 

0.054 

4.2-inch  mortar 

0.30 

0.063 

0.008 

0.061 

0.009 

0.013 

0.010  | 

2.36-inch  rocket 

0.30 

0.035 

0.010 

-0.027 

0.011 

0.018 

0.011  | 

2.36-inch  rocket 

0.30 

0.112 

0.016 

0.051 

0.010 

-0.099 

shotput 

0.35 

0.166 

0.006 

0.133 

0.008 

-0.098 

Two  points  are  clear  from  the  values  in  Table  4.  First,  the  inversion  process  is  very  robust,  with 
no  standard  deviations  larger  than  2  cm.  Second,  there  are  a  few  large  discrepancies  between  our 
inverted  positions  and  the  reported  ones.  All  offsets  are  below  our  target  value  of  25  cm,  with 
the  exception  of  T-002.  It  is  possible  that  some  items  have  drifted  or  settled  since  their  original 
emplacement. 

8.2  OBJECTIVE:  REDUCTION  OF  FALSE  ALARMS 

This  is  the  primary  measure  of  the  effectiveness  of  this  technology.  By  collecting  high-quality, 
precisely-located  data,  we  expect  to  be  able  to  discriminate  munitions  from  scrap  and  frag  with 
high  efficiency.  At  a  seeded  test  site  such  as  the  APG  standardized  test  site,  the  metric  for  false 
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alarm  elimination  is  straightforward.  We  prepared  a  ranked  dig  list  for  the  targets  we 
interrogated  with  a  dig/no-dig  threshold  indicated,  and  ATC  personnel  use  their  automated 
scoring  algorithms  to  assess  our  results. 


This  objective  was  successfully  met.  The  TEMTADS  surveyed  anomalies  detected  by  the 
MTADS  magnetometer  system  in  the  blind  grid  and  indirect  fire  areas.  For  the  blind  grid  test 
area,  the  discrimination  stage  results  are  summarized  in  Table  5  and  Table  6  (subsets  of  Table  6a 
of  Reference  8),  broken  out  by  munitions  type  and  emplacement  depth.  For  the  Indirect  Fire 
Test  Area,  the  discrimination  stage  results  are  summarized  in  Table  7  and  Table  8  (subsets  of 
Table  6c  of  Reference  8),  broken  out  by  munitions  type  and  emplacement  depth.  The 
discrimination  stage  probability  of  detection  (Pddlsc)  is  defined  as  the  number  of  correctly 
identified  munitions  divided  by  the  number  of  emplaced  munitions,  and  the  corresponding 
probability  of  false  positive  (Pfpdlsc)  is  the  number  of  clutter  items  incorrectly  identified  as 
munitions  divided  by  the  number  of  emplaced  clutter  items. 

Table  5.  TEMTADS  blind  grid  test  area  Pddise  results. 


r>  disc 

fd 

All  Types 

105  mm 

81/60  mm 

37/25  mm 

Munitions  scores 

0.97 

0.93 

0.97 

1.00 

0  to  4D  a 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

8D  to  12D 

0.67 

0.67 

0.00 

1.00 

a  Burial  depths  are  reported  in  units  of  the  munitions’  outer  diameter,  or  D 


Table  6.  TEMTADS  blind  grid  test  area  Pfpd,st  results. 


r»  disc 

All  Masses 

0  to  0.25  kg 

>0.25  to  1  kg 

>1  to  10  kg 

All  depths 

0.01 

0.02 

0.00 

0.00 

■  0  to  0.15m 

0.01 

0.02 

0.00 

0.00 

;  0.15  to  0.3m 

0.00 

0.00 

0.00 

0.00 

0.3  to  0.6m 

N/A 

N/A 

N/A 

N/A 

Table  7.  TEMTADS  indirect  fire  test  area  Pddise  results. 


ry  diSC 

"d 

All  Types 

105  mm 

81/60  mm 

37/25  mm 

Munitions  scores 

0.92 

0.93 

0.93 

0.91 

By  Density 

High 

0.88 

0.92 

0.91 

0.80 

Medium 

0.94 

0.97 

0.89 

0.97 

Low 

0.94 

0.90 

0.97 

0.94 

By  Depth 

0  to  4D 

0.96 

0.94 

0.97 

0.97 

4D  to  8D 

0.92 

0.94 

0.92 

0.86 

8D  to  12D 

0.72 

0.75 

0.78 

0.67 
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Table  8.  TEMTADS  indirect  fire  test  area  PfPdlst  results. 


r>  disc 

'in 

All  Masses 

0  to  0.25  kg 

>0.25  to  1  kg 

>1  to  10  kg 

All  depths 

0.04 

0.03 

0.02 

0.11 

0  to  0.15m 

0.04 

0.04 

0.02 

0.13 

0.15  to  0.3m 

0.04 

0.00 

0.06 

0.06 

0.3  to  0.6m 

0.08 

0.00 

0.00 

0.20 

Discrimination  efficiency  (E)  and  false  positive  rejection  rate  (Rfp)  measure  the  effectiveness  of 
the  discrimination  stage  processing.  The  goal  of  discrimination  is  to  retain  the  greatest  number 
of  munitions  detections  from  the  anomaly  list,  while  rejecting  the  maximum  number  of 
anomalies  arising  from  non-munitions  items.  Efficiency  measures  the  fraction  of  detected 
munitions  retained  after  discrimination,  while  the  rejection  rate  measures  the  fraction  of  false 
alanns  rejected.  The  measures  are  defined  relative  to  the  number  of  munitions  items  or  the 
number  of  clutter  items  that  were  actually  detected  by  the  sensor.  The  results  for  the  blind  grid 
and  indirect  fire  test  areas  are  summarized  in  Table  9  and  Table  10,  from  Tables  7a  and  7c  of 
Reference  8.  Perfonnance  levels  are  shown  at  two  specific  operating  points  on  the  ROC  curve: 
one  at  the  point  where  no  decrease  in  Pd  is  incurred  and  the  other  at  the  operator-selected 
operating  point  or  threshold.  In  the  blind  grid,  98%  of  the  emplaced  munitions  items  were 
detected.  Of  these,  99%  (97%  of  the  emplaced  munitions)  were  correctly  classified  at  our 
selected  operating  point,  with  a  corresponding  false  positive  rejection  rate  of  99%.  Moving  out 
the  ROC  to  the  point  where  100%  of  the  detected  munitions  items  are  correctly  classified 
reduces  the  false  positive  rejection  rate  to  95%.  In  the  indirect  fire  area,  94%  of  the  emplaced 
munitions  were  detected,  and  92%  of  the  emplaced  munitions  were  correctly  classified,  resulting 
in  a  discrimination  efficiency  of  98%.  The  corresponding  false  positive  rejection  rate  was  92%. 

Table  9.  TEMTADS  blind  grid  test  area  efficiency  and  rejection  rates. 


Efficiency  (E) 

False  Positive  Rejection  Rate 

At  operating  point 

0.99 

0.99  ! 

With  no  loss  of  Pd 

1.00 

0.95 

Table  10.  TEMTADS  indirect  fire  test  area  efficiency  and  rejection  rates. 


Efficiency  (E) 

False  Positive  Rejection  Rate 

At  operating  point 

0.98 

0.92  ! 

With  no  loss  of  Pj 

1.00 

0.58  ! 

8.3  OBJECTIVE:  LOCATION  AND  DEPTH  ACCURACY 

An  important  measure  of  how  efficiently  any  required  remediation  will  proceed  is  the  accuracy 
of  predicted  location  of  the  targets  marked  to  be  dug.  Large  location  errors  lead  to  confusion 
among  the  UXO  techs  assigned  to  the  remediation,  costing  time  and  often  leading  to  removal  of 
a  small,  shallow  object  when  a  larger,  deeper  object  was  the  intended  target.  The  average  error 
and  deviation  in  both  horizontal  axes  was  computed  for  the  items  that  are  selected  for  excavation 
during  the  validation  phase  of  the  demonstration.  We  provided  an  estimated  position  for  all 
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targets  we  interrogated  to  the  appropriate  personnel  at  each  site,  and  they  used  their  scoring 
algorithms  to  assess  our  results. 


This  objective  was  not  successfully  met  in  full.  For  the  APG  demonstration,  the  location 
accuracy  of  fit  parameters  generated  from  the  TEMTADS  array  data  are  given  in  Table  11  and 
Table  12,  taken  from  Tables  9a  and  9c  of  Reference  8.  Horizontal  errors  are  not  calculated  for 
the  blind  grid. 

Table  11.  TEMTADS  blind  grid  test  area  location  error  and  standard  deviation. 


Mean 

Standard  Deviation 

(m) 

(m) 

Northing 

N/A 

N/A  i 

Easting 

N/A 

N/A 

Depth 

0.02 

0.04  1 

Table  12.  TEMTADS  indirect  fire  test  area  location  error  and  standard  deviation. 


Mean 

Standard  Deviation 

(m) 

(m) 

Northing 

0.01 

0.05 

Easting 

0.01 

0.05 

Depth 

0.00 

0.06 

For  the  former  Camp  SLO  demonstration,  the  average  northing  position  error  for  all  measured 
data  was  1.5  cm,  while  the  average  easting  position  error  was  3  cm.  The  standard  deviations, 
however,  were  larger  than  desired,  with  both  values  about  25  cm.  Excluding  those  anomalies  for 
which  multiple  targets  were  found  produces  negligible  improvement.  We  suspect  the  higher 
values  are  due  to  the  large  number  of  small,  low  SNR  clutter  items,  which  result  in  greater 
uncertainty  in  both  the  measured  and  fitted  values.  Indeed,  if  we  restrict  ourselves  to  those 
anomalies  that  we  classified  as  “likely  UXO,”  the  northing  and  easting  standard  deviations  drop 
to  7  cm  and  5  cm,  respectively.  The  average  depth  error  for  all  measured  data  was  2.5  cm.  The 
standard  deviation  was  a  bit  larger  than  desired,  with  a  value  of  14  cm.  Excluding  those 
anomalies  for  which  multiple  targets  were  found  and  restricting  the  analysis  to  those  anomalies 
classified  as  “likely  UXO”  to  exclude  low  SNR  clutter  items,  the  standard  deviation  reduces  to 
7  cm. 
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9.0  COST  ASSESSMENT 


The  costs  for  a  3000  anomaly  cued  survey  using  the  TEMTADS  are  detailed  in  Table  13.  The 
costs  associated  with  the  fonner  Camp  SLO  demonstration  were  used  as  the  basis  for  this 
assessment.  The  costs  reported  here  do  not  include  any  costs  associated  with  the  reconnaissance 
data  that  was  used  to  identify  and  create  the  cued  list  of  anomalies.  Implicit  in  our  assessment  is 
that  that  the  3000  anomalies  are  bounded  within  an  area  of  roughly  25  acres,  or  120 
anomalies/acre.  Survey  time  would  increase  incrementally  as  the  anomaly  density  decreased  as 
transit  time  between  anomalies  increased.  All  costs  are  estimated  in  fiscal  year  (FY)  2009 
dollars. 


Table  13.  Summary  of  costs  for  a  25-acre,  3000  anomaly  TEMTADS  survey. 


Cost  Category 

Sub-Category 

Cost 

Mobilization  costs 

Preliminary  site  visit 

$6500 

Test  plan  preparation 

$10,250 

Equipment  prep  and  packing 

$9750 

53 -ft  trailer  transportation 

$5800 

Analysts  set-up 

$10,000 

Outbound  travel  for  3  personnel 

$2250 

Subtotal 

$44,550 

Logistics 
(if  required) 

Establish  GPS  control  points 

$2000 

Delivery/removal  of  logistics  items 

$2000 

Rental  of  logistics  items/ fuel 

$6000 

Materials 

$6500 

Subtotal 

$16,500 

Operating  costs 
(4-week  Survey) 

Supervisor 

$19,000 

On-site  analyst 

$31,800 

Vehicle  operator 

$12,750 

Field  technician  (2  each) 

$17,250 

Per  diem  (3  personnel  x  4  weeks  k  $1250) 

$15,000 

Rental  vehicles 

$8650 

System  maintenance 

$13,000 

Sensor  repair 

$21,600 

Subtotal 

$139,050 

Analysis  &  reporting 

Data  reduction  to  anomaly  list 

$28,000 

Demonstration  data  report 

$16,750 

Subtotal 

$44,750 

Demobilization  costs 

Equipment  unpacking 

$3500 

53 -ft  trailer  transportation 

$5800 

Inbound  travel  for  3  personnel 

$2250 

Subtotal 

$11,550 

Total  Cost 

$256,400 

9.1  COST  MODEL 

A  cost  model  was  constructed  from  the  costs  associated  with  the  TEMTADS  survey  of  the 
former  Camp  SLO  demonstration  site.  Cost  categories  were  mobilization,  logistics  (if  required), 
field  work,  demobilization,  data  analysis,  and  reporting.  Based  on  the  demonstrated  production 
rate  for  the  fonner  Camp  SLO  demonstration,  4  weeks  of  field  work  would  be  required.  Based 
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on  the  production  rate  for  the  APG  demonstration,  the  SLO  production  rate  is  a  conservative 
lower  bound.  This  model  does  not  include  the  cost  of  anomaly  classification  and  producing  a 
prioritized  dig  list. 

9.2  COST  DRIVERS 

Two  factors  were  expected  to  be  strong  drivers  of  cost  for  this  technology  as  demonstrated.  The 
first  is  the  number  of  anomalies  that  can  be  surveyed  per  day.  Higher  productivity  in  data 
collection  equates  to  more  anomalies  investigated  for  a  given  period  of  time  in  the  field.  The 
time  required  for  analyzing  individual  anomalies  can  be  significantly  higher  than  for  other  more 
traditional  methods  and  could  become  a  cost  driver  due  to  the  time  involvement.  The  thoughtful 
use  of  available  automation  techniques  for  individual  anomaly  analysis  with  operator  QC  support 
can  moderate  this  effect. 

9.3  COST  BENEFIT 

The  main  benefit  to  using  a  UXO  classification  process  is  cost-related.  The  ability  to  reduce  the 
number  of  nonhazardous  items  that  have  to  be  dug  or  dug  as  hazardous  directly  reduces  the  cost 
of  a  remediation  effort.  The  additional  infonnation  for  anomaly  classification  provided  by  the 
TEMTADS  array  provides  additional  information  for  the  purposes  of  anomaly  classification.  If 
there  is  buy-in  from  the  stakeholders  to  use  these  techniques,  this  information  can  be  used  to 
reduce  costs. 

As  an  example  of  the  potential  cost  benefit  of  using  this  technology  on  an  actual  cleanup,  an 
example  scenario  is  presented.  The  two  demonstrations  discussed  in  this  report  were  both  of 
short  duration  with  a  small  number  of  anomalies  to  capitalize  mobilization  costs  across. 
Therefore,  considering  only  the  field  work  and  data  analysis  costs  given  in  Table  13,  a 
reasonable  estimate  of  the  cost/anomaly  for  deploying  the  TEMTADS  array  long  tenn  is 
$6 1/anomaly.  The  staffing  level  assumed  in  Table  13  is  appropriate  for  a  new  research  and 
development  project  demonstration  and  could  be  reduced  to  support  sustained  production 
operations. 

Similarly,  the  cost  of  fielding  an  appropriately  certified  UXO  dig  team  without  mobilization 
costs  can  range  between  $37,000  and  $50,000  per  week  (FY  2010  dollars).  Assuming  that  the 
team  can  clear  between  310  and  420  anomalies  a  day,  the  cost  to  dig  an  anomaly  is  $90  - 
$160/anomaly.  Assuming  that  1%  of  the  items  dug  are  in  fact  UXO,  the  remediation  of  those 
UXO  must  be  accounted  for  and  a  remediation  cost  of  $1000/UXO,  the  average  cost  per  dig 
would  range  from  $100  -  $170/anomaly. 

Finally,  for  the  example,  assume  a  hypothetical  actual  cleanup  site  with  10,000  anomalies  to  be 
cleared.  Using  the  above  analysis,  the  cost  of  the  cleanup  with  all  anomalies  dug  would  range 
from  $1,000,000  to  $1,700,000  total.  If  one  assumes  that  the  TEMTADS  classifies  the 
anomalies  sufficiently  well  to  reduce  the  number  of  actual  digs  required  to  10%  of  the  original 
number,  the  combined  cost  of  the  TEMTADS  survey  and  the  resultant  digging  drops  to  $713,000 
-  $783,000  total,  for  a  potential  savings  of  29  -  54%  overall. 
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10.0  IMPLEMENTATION  ISSUES 


Implementation  issues  for  this  system/technology  fall  into  two  categories:  operational  concerns 
and  data  quality/analysis  issues.  In  terms  of  operational  concerns,  the  TEMTADS  as 
implemented  is  a  large,  vehicle-towed  system  that  operates  best  in  large,  open  areas.  As  seen  at 
the  fonner  Camp  SLO  demonstration,  it  is  possible  to  operate  the  system  in  rocky  terrain  with 
grades  approaching  20%,  but  at  reduced  operating  capacity  and  increased  system  wear.  Smaller 
versions  of  the  system  are  currently  under  development  under  several  ESTCP  and  SERDP 
projects  to  address  these  concerns.  The  goal  is  to  design  and  field  units  more  amenable  to 
operation  in  more  confined  terrain  and  topology  and  smaller  tow  vehicles/man-portable  and 
handheld  operation.  Another  serious  limitation  will  be  anomaly  density.  For  all  sensors,  there  is 
a  limiting  anomaly  density  above  which  the  response  of  individual  targets  cannot  be  separated. 
We  have  chosen  relatively  small  sensors  for  this  array,  which  should  help  with  this  problem,  but 
we  cannot  eliminate  it.  Based  on  experiments  at  our  test  pit  at  Blossom  Point,  the  results  of  this 
demonstration,  and  work  done  on  the  fonner  Camp  Sibert  data  sets,  anomaly  densities  of  300 
anomalies/acre  or  higher  would  limit  the  applicability  of  this  system  as  more  than  20%  of  the 
anomalies  would  have  another  anomaly  within  a  meter. 

In  tenns  of  data  quality,  one  pays  a  small  penalty  in  signal  amplitude  to  use  the  smaller 
TEMTADS  sensor  coil  as  compared  to  other  systems  such  as  a  high-power  EM61  Mkll.  The 
dramatically  improved  electrical  perfonnance  of  these  new  sensors  helps  counterbalance  this 
issue,  particularly  in  the  ability  to  perfonn  better  averaging  (or  stacking).  However,  one  needs  in 
place  robust  data  QC  techniques  to  know  when  to  employ  these  capabilities  in  an  efficient 
manner. 
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